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FOREWORD 


The  US  Army  Research  Institute  (ARI)  in  cooperation  with 
TRADOC  and  its  schools,  performs  research  and  development  on  ways 
to  achieve  more  cost-effective  training.  In  1967  ARI  joined  with 
TRADOC  and  the  US  Army  Ordnance  Center  and  School  (USAOCS)  in  a 
partnership  at  Aberdeen  Proving  Ground  (APG),  Maryland  to  identify 
and  solve  maintenance  training  problems.  The  partnership  was 
defined  by  a  memorandum  of  understanding  (MOU)  entitled 
"Establishment  of  a  Training  Technology  Field  Activity  (TTFA)  at 
the  USAOC&S" ,  and  dated  10  May  1987. 

The  present  report  is  the  final  result  of  that  partnership. 
The  work  was  carried  out  as  part  of  Task  344  by  a  member  of  the 
Logistics  Training  Technologies  Technical  Area  of  the  Training 
Research  Laboratory  in  order  to  provide  a  concise  summary  of  all 
the  TTFA-Aberdeen  projects  done  under  task  344. 

This  and  other  products  of  the  TTFA  were  briefed  to  the 
Deputy  Assistant  Comma adar t ,  USAOC&S,  in  October,  1986.  It  will 
be  used  to  help  develop  model  training  instruction  and  plan 
additional  TTFA  proje:ts  at  USAOC&S.  Its  value  was  recognized  by 
MG  Ball,  Commandant,  to  USAOC&S,  in  a  letter  of  appreciation  to 
the  author. 


EDGAR  M.  JOHNSON 
Technical  Director 
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CONTRIBUTIONS  OF  THE  ARMY  RESEARCH  INSTITUTE  TO  THE  TRAINING 
TECHNOLOGY  FIELD  ACTIVITY- -ABERDEEN 


EXECUTIVE  SUMMARY 


Requirement : 

Summarize  research  on  maintenance  performance  problems  and 
describe  preliminary  efforts  by  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  to  address  the  prob¬ 
lems  through  the  Training  Technology  Field  Activity- -Aberdeen . 


Procedure : 

The  author  reviewed  reports  of  eleven  studies  conducted 
during  the  past  25  years  and  five  reports  prepared  by  ARI  under 
the  TTFA -Aberdeen  Program.  He  distilled  data  and  conclusions 
from  the  reports  and  presented  them  in  a  series  of  tables  and 
figures . 


Findings : 

1.  Failure  rates  among  both  novice  and  experienced  vehicle 
repairers  are  dramatically  high  and  should  be  a  matter  of  serious 
concern  to  the  Army,  e.g.,  the  false  removal  rate  (goods  parts 
taken  out)  is  about  U2X  .  Yet,  in  spite  of  25  years  of  evidence 
accumulated  by  the  Army  itself,  the  problems  remain. 

2.  The  TTFA- Aberdeen  provides  a  significant  opportunity  to 
begin  to  systematically  address  maintenance  performance 

def iciencies . 

3.  Under  the  TTFA  program,  the  Army  Research  Institute  has 
generated  some  preliminary  recommendations  and  guidelines  for 
developing  training  which  will  substantially  Improve  skill  acqui¬ 
sition,  retention,  and  transfer  for  vehicle  repairers. 

4.  These  recommendations  and  guidelines  are  presented  in 
the  report  in  tables  and  figures. 


vii 


Use  of  Findings: 

1.  The  findings  can  be  used  immediately  to  improve  training 
for  vehicle  repairers  at  the  U.S.  Army  Ordnance  School. 

2.  Although  targeted  to  vehicle  maintenance  training,  the 
findings  can  be  generalized  to  other  types  of  training. 

3.  The  results  described  in  this  report  can  provide  a 
spring  board  for  continuing  research,  development,  and  training 
technology  transfer  under  the  TTFA- Aberdeen  Program. 
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CONTRIBUTIONS  OF  THE  ARMY  RESEARCH  INSTITUTE  TO  THE 
TRAINING  TECHNOLOGY  FIELD  ACTIVITY — ABERDEEN 


1.0  INTRODUCTION 


1.1  This  summary  report  is  tbs  final  one  in  a  series  of  efforts 
for  ARI  in  the  two-year,  l'TFA  -Aberdeen  Program.  The  report 
explains  what  products  we  left  to  the  program  and  how  they  can  be 
used  by  the  Army.  I've  distilled  information  and  concluaiona  from 
5  earlier  reports.  But  to  appreciate  the  need  for  these  products 
and  their  potental  usefulness,  the  reader  needs  to  know  what  was 
done  before,  as  well  as  during,  the  tenure  of  the  TTFA  to  study 
maintenance  performance  deficiencies. 

1.2  During  the  last  23  years,  the  Army  hes  paid  for  at  least  10 
projects  to  assess  vehicle  maintenance  performance  (Table  1). 
Those  studies  have  repeatedly  shown  high  failure  rates  serosa  a 
variety  of  tasks,  conditions,  and  assesaisnt  techniques.  But 
until  the  advent  of  the  TTFA-Aberdeen ,  the  studies  appeared  to 
have  had  little  impact  on  how  the  Army  trains  its  maintalners.  In 
1979  ARI  did  successfully  develop  a  computer  technique  for 
tracking  maintenance  performance  (Maintenance  Performance  System), 
but  it  was  not  implemented  (Harper  et  al,  1979).  The  TTFA-Fort 
Knox  did  develop  a  computer-based  instruction  (CBI)  program  in 
maintenance  for  use  on  M I CCROT I CC I T ,  and  it  has  received  some 
adoption  (Craham  et  al,  1986).  Now  the  TTFA-Aberdeen  aa  shown  in 
this  report  has  provided  useful  products  which  are  being 
implemented  at  the  Ordnance  School. 

1.3  Significance  of  historical  perspective 

a.  Various  Army  elements  have  repeatedly  studied  and 
verified  the  same  set  of  maintenance  performance  shortfalls  for  25 
years.  Follow-up  needs  to  be  conducted  on  the  TTFA-Aberdeen 
result  to  ensure  that  the  total  TRADOC/ARI  expenditures  of  $2 
million  will  continue  to  pay  dividends. 

b.  The  problem  of  dealing  with  maintenance  performance 
deficiencies  is  especially  significant  because  the  Army  spends 
$22,000  per  year  (1988  dollars)  to  train  an  entry-level  mechanic 
(Unpublished  study  by  USA0CS).  The  total  cost  for  1500  students 
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per  year  in  the  63W10  program  at  Aberdeen  is  $33,000,000. 
Historically  high  failure  rates,  along  with  emerging  TTFA  results 
suggest  that  the  return  on  this  training  investment  can  be 
improved. 

c.  But  if  training  practice  changes  because  of  TTFA 
performance  analyses  and  recommendations,  the  TTFA  will  have  made 
history.  For  the  first  time  in  25  years  the  Army  will  have  paid 
major,  systematic  attention  to  its  well  documented  and  potentially 
serious  maintenance  performance  shortfall.  Section  2.0  (including 
Table  1)  summarizes  the  historical  evidence.  Section  3.0  outlines 
the  recent  analyses  and  recommendations. 


2.0  HISTORY  OF  ARMY  EVALUATIONS  OF  AUTOMOTIVE  MAINTENANCE 

PERFORMANCE 


2.1  In  1964  HumRRO  tested  63C  mechanics  as  they  diagnosed  and 
repaired  "bugged"  tanks,  trucks,  and  personnel  carriers  in 
simulated  work  environments  at  Fort  Knox  and  in  combat 
organizational  units  (Smith,  1964).  HumRRO  staff  recorded  the 
number  of  failures  to  complete  repair  jobs  and  the  number  of 
mis-diagnoses .  The  mechanics  failed  to  complete,  correctly  81%  of 
the  assignments  and  mis-diagnosed  60  %  of  the  problems  (Schurmac  & 
Kern,  1981).  In  a  similar  study,  but  one  using  field  observations 
of  actual  duties  in  organizational  units,  Applied  Sciences 
Associates  (ASA)  found  similar  results:  84%  of  the  assignments 
were  not  successfully  finished  (Schurman,  Joyce,  and  Porsche, 
1980). 

2.2  The  US  Army  Ordnance  &  Chemical  Center  (USAOCC),  in  1978, 
tested  the  performance  of  178  E1-E6  track  vehicle  mechanics  (63Cs) 
and  167  tank  automotive  r epa i r e rs ( 6 3Hs ) .  Each  soldier  was  tested 
on  8  tasks  involving  diagnosis,  replacement,  or  adjustment. 
Percent  of  63Cs  failing  ranged  from  53%  to  97%,  across  the  8 
tasks.  For  the  63Hs,  the  failures  ranged  from  56%  to  99%.  These 
figures  were  combined  with  data  on  time- to-diagnose  or  repair  and 
mean-time  between  failures  to  arrive  at  the  conclusion  that  "the 
potential  exists  for  equipment  availability  to  fall  to  fifty 
percent  or  less  for  such  critical  items  ss  the  main  battle  tank." 
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2.3  The  study  above  was  repeated  (aame  teaks  and  measures)  by 
USAOCC  on  the  National  Guard.  The  figurea  for  63Cs  were  78X  to 
100%  failures.  For  63Hai  56%  to  100%  The  same  teaks  ware 
evaluated  in  both  studies. 

2.4  The  Army  Research  Institute  collected  data  from  direct 
support  maintenance  shop  records  for  one  year  (Dressel  &  Shields, 
1979).  It  found  that  42%  of  the  parts  replaced  were  falsely 
identified  as  defective.  The  percent  of  labor  consumed  by  these 
false  removals  was  32%.  If  this  figure  is  at  all  representative 
of  the  labor  cost  of  deficient  maintenance,  it  would  be  a  matter 
of  some  concern  in  view  of  the  $15+  billion  per  year  which  the 
Army  spends  on  maintenance. 

2.5  In  1981  Rand  Corp.  surveyed  senior  Army  maintenance 
specialists  from  organizational,  direct  support,  and  general 
support  units  responsible  for  maintaining  a  variety  of  wheel  and 
track  vehicles  (Harz,  1981).  About  200  questions  probed  the 
existence  of  problems  and  ideas  for  solving  the  problems.  Of 
particular  interest  was  a  query  on  false  removals.  The  average 
estimate  of  rate  of  false  removal  was  36%,  which  is  quite  close  to 
the  Dressel  and  Shields  (1979)  more  objective  estimate  of  42%. 

2.6  The  Army  Research  Institute  (Kern  &  Hayes,  1983)  showed 
widespread  failures  among  236  organisational  wheel  vehicle 
mechanics,  on  5  different  vehicles.  71%  of  the  mechanics  made  one 
or  more  serious,  uncorrected  errors  involving  special  tools  or 
specifications,  per  maintenance  activity.  Experienced  mechanics 
used  the  technical  manual  for  1%  of  their  assignments,  while 
inexperienced  mechanics  used  the  manuals  for  12%  of  assignments. 
But  even  where  TMs  were  used,  failure  rates  were  still  very  high. 
Of  the  mechanics  who  did  use  TMs  (  >  r  other  sources  of 
information),  66%  made  one  or  more  procedural  errors  per  task 
compared  tr  79%  for  soldiers  who  did  not  consult  a  source  of 

info  rma  tic  i. 

2.7  In  1987,  the  TRADOC  Analysis  Command  (TRAC)  conducted  a  study 
similar  to  HumRRO's  1964  work  and  the  studies  by  USAOCC  in  1978. 
TRAC  tested  63U  end  63B  performance  on  5  tasks,  using  "bugged" 
wheel  vehicles.  The  task  failure  rates  ranged  from  85%  to  99%. 

2.8  Added  to  the  above  are  a  number  of  studies  done  by  ARI  under 
the  TTFA-Aberdeen  program.  These  will  be  described  in  Section 
3.0.  But  some  illustrative  results  are  shown  in  Table  1. 
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Additional  studies  of  basic  knowledges  and  skills  (BKS)  and  task 
performance  among  63W  students  and  mechanics  will  be  described  in 
a  contract  report  by  Applied  Sciences  Associates  (ASA),  in 
preparation.  However,  a  preliminary  result  of  the  BKS  testing  of 
41  soldiers  at  the  end  of  a  16-week  course  is  that  test  items  ware 
failed,  on  average,  by  56%  of  the  students.  The  written  test 
covered  6  diagnostic  tasks  and  1  replacement  task  selected  by  the 
instructors  as  critical. 
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Table  1.  Army  Studies  of  Automotive  Maintenance  Performance 


When 

Who 

Vhat 

Uhat  Found 

1964 

* 

HumRRO 

Tested  Track 

Vehicle  Mechanics 

81%  of  assignments 
Failed 

1978 

USAOCC 

Determined  Proficiency 
development  profiles 

63C  and  63H  MOSs 

Potential  for  50% 
down-time  on  Main 
Battle  tank 

1979 

ARI 

Collected  data  on  M551 
repairs 

42%  false  removals. 
(32%  of  total  labor) 

1980 

HumRRO 

Observed  TVMs 

84%  of  assignments 
failed 

1981 

Rand 

Surveyed  Org.,DS 
&  GS  personnel. 

37%  false  removals 

1983 

ARI 

Observed  Org.Mech., 

Ft.  Knox 

71%  of  assignments 
with  serious  errors. 
(Special  tools/spec) 

1987 

TRADOC 

(TRAC) 

Tested  63W&B 
performance  in 

FORSCOM  units 

85  to  99%  failure 
rates  across  5  tasks 

1988 

TTFA  (Ramsay) 

Summarized  end  of 
course  written 
test  data  by  class 

6  to  50%  failure 

rates  across  classes 

1988 

TTFA  (MacPher  ) 

Predicted  skill  decay 
with  ARI  model 

65%  of  soldiers  fail 
injector  pump  removal, 
after  1  month. 

1988 

TTFA 

(ASA) 

Tested  end  of  course 
BKS.  (contractor  test) 

56%  average  item 
failure  rate,  across 

72  items  for  7  tasks 
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3.0  ARI  CONTRIBUTIONS  TO  THE  TTFA- ABERDEEN  PROGRAM 


This  review  contains  five  sections:  3.1  Preliminary  Review 
of  the  63W  Course;  3.2  Handbook  for  Producing  Classroom  Vugrapha; 
3.3  Application  of  Skill  Retention  Model;  3.4  Development  of 
Guidelines  for  Improving  Skill  Retention  and  transfer;  and  3.5 
Application  of  Resource  Allocation  Modeling  to  Phase  2  of  the 
63W10  course. 


3.1  Preliminary  Review  of  the  63W10  Course  at  Aberdeen  Proving 
Ground  (Ramsay,  Kessler,  Mirabella,  &  Thoreson,  1988). 

a.  The  first  effort  of  the  TTFA  was  to  observe  Phases  1 
(BKS)  and  2  (task  training)  of  the  63U10  course  and  to  examine 
course  grades.  The  reason  for  doing  this  was  to  gain  first-hand 
knowledge  about  how  training  was  currently  being  conducted  and 
insights  into  how  it  might  be  improved.  Ve  maintained  daily  logs 
in  which  we  described  the  training  annex  by  annex  and  commented  on 
possible  training  problems  and  solutions.  An  annex  is  a  course 
segment  lasting  about  two  weeks. 

b.  To  obtain  quantitative  data,  we  also  compiled  failure 

rates  for  samples  of  end-of -annex  (EOA)  and  end-of -course  (EOC) 
written  tests.  Data  for  16  classes  in  Phase  1  are  shown  in  Table 
2.  These  are  failure  data  for  Annexes  A  through  G.  The  highest 
failure  rates  occurred  for  Annexes  C  and  E  at  15%  and  31  % 

respectively  of  all  the  students  in  those  annexes.  But  the 
relatively  low  rates  for  the  other  annexes  obscure  very  high 
failure  rates  for  some  classes  within  the  annexes.  For  example, 
in  the  basic  skills  annex,  34%  of  the  students  failed  in  one 
class,  though  the  average  across  all  classes  was  only  about  5%. 
Written  test  data  from  8  classes  in  Phase  2  are  shown  in  Table  3. 

c.  Use  of  Data. 

(1)  The  data  point  to  3  topics  for  which  revised  training 

may  be  especially  necessary:  Introduction  to  engine  systems, 

electrical  technology,  and  hydraulic  technology. 

(2)  The  data  plus  course  observations  led  to  a  set  of 
preliminary  recommendations  for  improving  the  63W10  Wheel  Vehicle 
Maintenance  Course.  One  recommendation  prompted  the  school  to 
revise  vugrapha  used  in  the  63W  Course  end  led  the  TTFA  to  develop 
a  handbook  on  how  to  produce  vugraphs. 
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d.  Recommendations  for  improving  the  63W10  course. 

(1)  Integrate  basic  skill  and  knowledge  training  (Phase  1) 
with  MOS  task  training  (phase  2).  The  separation  of  these  two 

phases  violates  a  fundamental  of  good  instruction  (TRADOC 
Pamphlet  600-11,  Page  37,  "Functional  Context"). 

(2)  Improve  the  visual  side  that  support  lectures,  using  the 
TTFA-deve loped  guidebook  on  visual  aids. 

(3)  Provide  study  guides  for  all  annexes  and  copies  of  TM  9- 
800  for  students  to  keep. 

(4)  If  actual  equipment  (AE)  examples  are  to  be  used  during 
lectures,  then  provide  one  unit  of  AE  per  2-3  students  and  pass 
them  out  beforehand  so  that  instruction  can  be  geared  to  the 
examp  les . 

(5)  Improve  the  quality  of  written  tests  by  using 
professional  test  development  techniques.  An  effort  is  needed 
which  will  produce  the  following. 


a . 

I  tern 

pools 

b. 

I  tern 

statistics 

c  . 

Alternate  forms 

d. 

Test 

statistics 

e  . 

Feedback  to  students 

(6)  Improve  the  quality  of  Phase  2  end-of -course  performance 
tests  by  testing  each  student  at  all  stations.  Develop  part-task 
testing  methods  so  that  it  will  be  feasible  to  test  all  students 
at  all  stations. 

(7)  Use  videotapes  of  experienced  mechanics  performing 
various  tasks  to  give  an  overview  and  to  illustrate  the  pace  and 
style  of  on-the-job  repair. 

(8)  Use  computerized  graphics  and  interactive  video  disc  to: 

help  trainees  grasp  difficult  or  abstract  concepts,  e.g. 
electrical  and  hydraulic  systems,  use  of  micrometers,  timing  of 
diesel  engines,  engine  cycles, 

reduce  student  to  actual  equipment  trainer  ratios,  and 

increase  student  centered  training. 

(9)  Add  training  on  difficult  maintenance  conditions,  e.g., 
"frozen"  bolts,  stripped  threads,  spilled  fluids,  accidents, 
f ires . 


(10)  Obtain  the  benefits  of  both  the  team  teaching  method  and 
in-place  experts  by  assigning  one  or  two  "advisors"  to  each  class 
instead  of  a  team  of  6-8  instructors. 
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Table  2.  Performance  Failure  Data  for  63U10  Course,  Phase  1 


Annex 

Fa i lures 

Mean  Rate 

(X) 

Min -Max 
(%) 

A. 

Common  Subject 

(40/560) 

7.4 

0.00-27.78 

B. 

Basic  Skills 

(30/557  ) 

5.38 

0.00-27.78 

C. 

Engine  Systems 

(166/530) 

31.32 

9 . 09-76.00 

D. 

Fuel,  Air,  Exhaust 

(35/416) 

8.14 

0.00-22.50 

E. 

Auto  Electricity 

(64/410) 

15.60 

2.94-36.36 

F. 

Brake  Systems 

(20/367  ) 

5.44 

0.00-27.27 

G. 

Steeving/ Suspension 

(10/368) 

2.71 

0.00-13.79 

Table  3.  Performance  Failure  Data  for  63W10  Course,  Phase  2 


Annex 

Fa i lure  s 

Mean  Rates 

(X) 

Min-Max 

(X) 

J-6  Hydraulics/Electricity 

N-l  End-of -Course  Written  Test 

(46/257) 

(55/257) 

17 . 89 

2  1.40 

6. 66- 30.00 

6.66- 50.00 
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3.2  Handbook  for  Producing  Classroom  Vugraphs  (Ramsay  and 
Mirabella  ,  1988) . 

a.  The  63W10  course  review  (Para.  3.1)  showed  that  classroom 
vugraphs  were  often  too  detailed,  not  clearly  tied  to  the 
teaching  point,  or  were  otherwise  difficult  to  understand.  To 
help  the  school  improve  its  visual  aids,  we  wrote  10  guidelines 
and  procedures  for  producing  vugraphs  from  already  published 
materials. 

b.  The  authors  analyzed  existing  vugraphs  and  figures  from 
the  course  textbook  (TM  9-8000)  for  deficiencies.  They  then 
distilled  a  set  of  rules  for  how  to  improve  visual  materials  and 
illustrated  the  rules  with  "before"  and  "after"  examples.  The 
rules  are  summarized  in  Table  4. 


Table  4.  Rules  for  Producing  Classroom  Vugraphs 

1.  Remove  all  unnecessary  material  from  around  the  figure. 

2.  Make  the  figure  reflect  the  main  teaching  point. 

3.  Eliminate  unnecessary  detail. 

4.  Reduce  the  number  of  call-outs  to  five  or  less. 

5.  Line  up  the  call-outs. 

6.  Make  call-outs  cue  words,  not  sentences  or  explanations. 

7.  Avoid  using  photographs. 

8.  Maintain  correct  spatial  w  i^.n  ta  tions  . 

9.  Check  for  accuracy. 

10.  Test  visual  aids  in  the  classroom. 
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3.3  Application  of  Skill  Retention  Model  to  63U10  Tasks 

(Macpherson,  Patterson,  &  Mirabella,  in  preparation) 

a.  The  purpose  of  this  project  was  to  supplement  data  on 
maintenance  performance  in  FORSCOM  units,  collected  for  TTFA,  by 
Applied  Sciences  Associates  (ASA).  We  applied  ARI's  skill 
retention  model  (Rose  et  al,  1983).  This  is  an  inexpensive  way  to 
estimate  how  quickly  skills  will  decay  for  any  specified  task 
which  is  not  practiced  after  training.  The  model,  which  provides 
pure  measures  of  forgetting,  can  help  interpret  data  on 
performance  actually  observed  in  units.  Ue  felt  that  the  model 
could  also  be  a  useful  tool  at  the  US  Army  Ordnance  School 
(USAOCS)  for  helping  course  developers  prioritize  t  a  n V i .  Ue 
wanted  to  try  it  out  on  USAOCS  subject-matter  experts  vSMEs)  to 
see  whether  they  could  use  it  routinely. 

b.  The  model  includes  a  set  of  10  scales  which  SMEs  use  to 
make  Judgements  about  task  characteristics,  e.g.  how  many  steps 
does  the  task  contain;  what  is  the  quality  of  the  Job  aids  used 
with  the  task.  The  SME  answers  each  question  (scale).  Scale 
values  attached  to  the  answers  are  then  added  to  provide  a  total 
which  is  used  to  enter  a  table.  From  this  table,  a  forgetting 
curve  is  generated.  Where  several  SMEs  make  different  Judgements 
that  cannot  be  reconciled,  different  retention  curves  are 
generated.  Multiple  curves  were  prepared  for  three  tasks: 

"  engine  cranks,  doesn't  start",  "remove/  replace  steering  gear", 
and  r em o v e / r e p la c e  injector  pump".  Which  of  the  alternative 
curves  should  you  use?  Look  for  some  outside  evidence  that 
supports  one  or  the  other.  Otherwise,  it's  probably  best  to  use 
the  lower  estimates  of  retention. 

c .  Results: 

(1)  Figures  1  and  2  show  forgetting  curves  for  tasks 
included  in  the  ASA  study.  Figures  1  and  2  shows  data  for 
diagnostic  tasks  which  ASA  had  originally  planned  to  observe. 
Figure  2  shows  the  remove  and  replace  tasks  it  was  actually  able 
to  observe.  From  these  curves  we  can  prioritise  tasks  from  most 
easily  forgotten  to  most  easily  remembered.  In  fact  TRADOC  PAM 
351-4(T)  specifically  tells  you  to  do  so  as  part  of  any  front-end 
analysis.  The  model  la  a  quantitative  way  to  put  the  TRADOC 
guidance  into  practice.  This  guidance  would  lead  us  to  put  high 
priority  for  training  "remove/replace  injector  pump"  and 
" remove / re p lace  steering  gear". 
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(2)  The  model  not  only  suggests  what  tasks  may  be  problems, 
but  also  why  they  are  problems,  and  how  to  Improve  performance. 
For  example  if  a  task  is  given  a  poor  retention  rating  because  its 
technical  manuals  are  poorly  written,  the  indicated  solution 
might  be  to  prepare  easy-to-use  job  aids.  Low  ra  Lings  on  the 
other  scales  would  suggest  other  ways  to  improve  performance. 

(3)  Furthermore,  with  the  help  of  the  scale  values,  we  can 
even  estimate  how  much  improvement  would  result  from  changes 
suggested  by  the  scales.  For  example,  good  quality  Job  aiding 
could  increase  retention  of  Injector  pump  skills  from  2  weeks  to  2 
months.  If  we  combine  that  information  with  the  cost  of  the 
improvement,  we  can  estimate  "how  much  a  pound  of  training  is 
worth".  AR1  reports  by  Adams  (1983)  and  Thoreson  (1988)  contain 
useful  ideas  and  procedures  on  how  to  do  cost-effectiveness 
trade-offs.  The  Skill  Retention  Model  supplies  the  performance 
data  we  have  always  lacked  to  do  such  trade-offs. 
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Months  since  last  successful  performance 

Fig  1.  Predicted  engine  diagnostic  performance 
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Fig  2.  Predicted  remove/replace  performance 


d.  Value  of  the  Skill  Retention  Model.  The  model  is 
particularly  useful  because  it  can  generate  performance 
predictions  quickly  and  inexpensively.  We  were  able  to  collect 
data  for  10  63W10  tasks  in  two  days.  The  empirical  study  by  ASA 
will  generate  data  for  5  tasks,  at  a  contract  cost  of  $200,000, 
over  9  months.  The  empirical  study  was  essential  for  TTFA 
purposes  and  could  not  have  been  avoided.  But  because  field 
studies  are  so  expensive  and  time  consuming  they  cannot  be  used 
routinely  for  front-end  analysis.  The  Skill  Retention  Model  can, 
in  production  use,  process  tasks  at  the  rate  of  one  per  hour  and 
therefore  provides  a  feasible  quantitative  tool  to  support  the 
task  prioritization  called  for  by  TRAD0C  PAM  351-4(t). 

e.  Project  Conclusions:  The  Skill  Retention  Model  can: 

(1)  be  administered  to  SMEs  at  the  Ordnance  School, 

(2)  flag  tasks  which  are  easily  forgotten, 

(3)  show  task  features  which  "cause"  skill  decay, 

(4)  point  to  training  fixes, 

(5)  support  cos  t- training  effectiveness  analysis. 


3.4  Development  of  Guidelines  for  Imp  r  ov  i.ig  Skill  Retention  & 
Transfer  Among  63W10  Repairers  (Mirabella,  Macpherson,  & 

Patterson,  in  preparation)  . 

a.  The  project  was  aimed  at  supporting  an  evaluation  of 
maintenance  performance  in  FORSCOM  units.  That  evaluation,  done 
under  contract  by  ASA,  was  designed  to  uncover  performance 
def ic ieucies ,  but  not  necessarily  to  provide  training  solutions. 
Therefore,  we  attempted  to  produce  a  general  set  of  training 
development  principles  based  on  a  review  of  research  literature, 
TRADOC  PAM  600-11,  and  the  TRAD0C  sponsored  Harless  Workshop. 
Furthermore,  we  expected  to  apply  those  principles  to  a  63W10 
task,  using  "before"  and  "after"  examples  in  the  manner  of  the 
vugraph  handbook.  We  were  not  able  to  pursue  the  application 
because  AR1  withdrew  from  the  TTFA-Aberdeen.  But  we 
recommend  that  ASA  consider  the  principles  in  completing  its 
last  contractual  task  which  is  to  develop  a  training  solution 
strategy  for  deficiencies  found  in  its  front-end  analysis. 
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b.  Results:  We  generated  10  principles,  a  large  number  of 
corollaries,  explanations  for  each,  and  guidance  on  how  to  put 
the  principles  into  practice.  The  principles  are  summarized  in 
Table  5 . 

c.  Project  Concluaions: 

(1)  Maintenance  performance  deficiency  is  a  significant,  but 
predictable  problem  in  the  Army, 

(2)  Techniques  exist  to  reduce  these  deficiencies  to  a 
negligible  amount, 

(3)  The  techniques  are  fully  compatible  with  TRADOC  REG 
71-2,  PAM  600-11,  and  PAM  351-4(T), 

(4)  They  are  condensed  in  the  principles  generated  by  this 
project , 

(5)  They  are  not  currently  being  uaed,  or  they  are  being 
used  incorrectly, 

(6)  To  use  them  effectively  we  need  to  shift  from  a  lecture/ 
content-oriented  style  of  training  to  one  that  is 
workshop/ task-oriented. 


Table  5.  Principles  for  Improving  Skill  Acquisition,  Retention 
and  Transfer  for  63Ws. 


1.  Identify  and  prioritize  problem  tasks.  Use  the  ARI  Skill 
Retention  Model,  along  with  any  available  performance  evaluation 
da  ta . 

2.  Id  entify  problem  steps  within  tasks.  Tasks  are  often  failed 
because  one  or  two  steps  are  difficult.  Give  these  special 
a  tten  tion . 

3.  Integrate  Basic  Knowledge  and  Skills  with  task  training,  i.e. 
Phase  l  and  2  of  the  63U10  Course.  Doing  so  would  conform  to  the 
principles  of  functional  context  training  specified  in  TRADOC  PAM 
600-11. 
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4.  Replace  the  current  lecture/content  oriented  philosophy  with  a 
workshop/ task  orientation.  Thia  ia  alao  consistent  with  the 
functional  context  principles  specified  in  TRADOC  PAM  600-11*  (See 
also  the  AR1  Guidebook  for  the  Development  of  Army  Training 
Li tera ture )  . 

5.  Use  the  basic  rule  of  good  Instruction  from  the  TRADOC- 
sponsored  Harless  Workshop:  the  5  Ps . 

a.  Prime:  Expla in/demons tra te/ check  for  comprehension 

b.  Prompt:  Give  part-task  training  with  help 

c.  Perform:  Give  part-task  training  with  reduced  help 

d.  Isolated  Practice:  Give  whole  task  training 

e.  Integrated  Practice:  Combine  whole  task  with  related 
tasks 

6.  Raise  test  standards  and  improve  performance  testing.  The  more 
effectively  soldiers  learn,  the  slower  they  forget.  Raise  the 
passing  grade  from  70%  to  85%,  and  increase  remedial  training 
accordingly. 

7.  Sequence  training  from  general  to  specific  information, 
concrete  experience  to  abstract  ideas,  and  familiar  to  unfamiliar 
ma  ter ia 1 . 

8.  Use  specialized  part-task  techniques,  e.g 

a.  Break-up  long  tasks 

b.  Provide  extra  practice  for  hard  steps  like  difficult 
decisions 

c.  Use  "deep  processing",  e.g.  analogies,  mnemonics,  probe 
ques  tions . 

9.  Train  for  transfer  to  new  equipment  as  well  as  retention,  e.g. 

a.  Train  to  proficiency  (i.e.  to  standards)  on  a  particular 

task 

b.  Repeat  practice  1  to  3  times  beyond  proficiency 
(over  train) . 

c.  Spread  practice  over  3  to  5  days 

10.  Improve  job  aiding/TM  training. 

a.  Focus  on  multi-TM  tasks  (these  are  most  difficult) 

b.  Develop  easy-to-use  Job  aids  for  most  difficult  tasks  or 
steps 
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3.5  Application  of  Resource  Allocation  Modeling  to  Phase  2  of  the 
63W10  Course  (Kessler,  in  preparation) . 

e.  During  our  observation  of  the  63U  Course,  Phese  2,  we 
noticed  thst  'actual  equipment  trainers  (AETs)  were  idle  about  50% 
of  the  time.  Yet  the  student  to  AET  ratio  was  often  high,  some 
students  did  not  participate  fully  in  group  exercises,  and  tasks 
were  practiced  only  once.  We  saw  an  opportunity  for  using  AETs 
much  more  efficiently.  The  TTFA-Ft  Rucker  had  reported  success 
using  a  computer  -based  model  to  allocate  (i.e.  schedule) 
training  resources  efficiently.  Ue  decided  to  see  whether  the 
Rucker  model  could  be  used  at  the  Ordnance  School. 

b.  We  saw  demonstrations  of  the  Rucker  Model  and  reviewed 

its  documentation.  We  also  examined  course  scheduling  problems  at 
Rucker.  Conclusion:  the  specific  Rucker  model  is  not  exportable 

to  USAOCS  because  scheduling  problems  are  unique  at  the  Aviation 
School.  Course  modules  there  are  sequence  dependent,  constrained 
by  weather,  and  involve  match-ups  between  instructors,  students, 
and  specific  AETs  (helicopters).  But  M1CR0SAXNT,  a  simulation 
program  underlying  the  Rucker  model,  could  be  used  at  Aberdeen, 
not  only  to  improve  efficiency  in  training,  but  also  to  automate 
portions  of  the  scheduling  task  in  Phase  2  of  the  63W10  course. 
MICROSAINT  is  a  high  level  programming  language  which  simulates 
networks  of  events  or  tasks.  It  is  designed  to  solve  resource 
allocation  and  scheduling  problems. 

c.  Figure  4  shows  how  we  used  MICROSAINT  to  simulate  Phase  2 
of  the  63W10  course.  We  told  the  computer  which  classes  were 
entering  Phase  2,  and  in  what  sequence.  We  also  identified  the 
annexes  and  times  to  complete  them.  We  instructed  the  computer  to 
assign  each  incoming  class  to  an  "empty"  annex,  and  then  reassign 
it  to  the  next  one  available  after  the  first  one  was  completed. 
The  result  of  the  simulation  was  a  chronological  list  of  class 
assignments  with  begining  and  ending  times  for  each  annex. 

d.  Unfinished  business.  We've  tried  to  show  an  opportunity 
for  making  63W10  training  more  cost-effective  and  to  demonstrate 
how  to  do  so.  To  finish  this  work,  a  number  of  things  have  to  be 
done . 
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(1)  The  output  format  of  our  simulation  is  not 
user-friendly.  It  consists  of  a  list  of  class  by  annex 
assignments.  The  information  needs  to  be  formatted  as  a  calender 
of  events,  so  that  the  user  can  clearly  see  who  goes  where,  and 
when.  This  may  require  the  use  of  additional  programming 
capability,  e.g.  a  spread  sheet  or  a  data  base. 

(2)  MICROSAINT  is  not  easy  to  use.  Someone  needs  to  create  a 
job  aid  for  routine  users  not  steeped  in  the  software. 

(3)  But  firat  someone  who  knows  both  MICROSAINT  and  the 
63W10  scheduling  requirements  needs  to  program  and  experiment  with 
various  rules  and  constraints  to  optimise  the  uee  of  AETs  in  Phaee 
2.  e.g  sub-grouping  of  clesses  to  reduce  claee  else,  duration  of 
lessons  within  annexes,  sequence  dcpendenciee  among  lcaaona  or 
annexes,  number  of  AETs,  number  of  instructore.  For  example, 
auppose  you  told  the  computer  to  divide  Incoming  classea  into 
groups  and  then  to  randomly  assign  groups  to  "empty"  lessons  or 
annexes,  what  effect  would  thet  have  on  AET  idle  time  and  atudent 
to  AET  ratio? 
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d.  Value  of  M1CR0SAXNT.  Given  a  50%,  or  even  smaller  idle 
time  for  AETs  and  a  price  tag  of  $22,000  per  atudent  in  the  63V10 
Course,  the  pay-off  for  more  efficient  use  of  training  resources 
could  be  very  substantial.  With  the  time  saved,  the  Ordnance 
School  could  apply  some  of  the  methods  recommended  by  other  TTFA 
projects  to  improve  skill  acquisition,  retention,  and  transfer. 


4.0  CONCLUDING  COMMENTS 

4.1  The  TTFA  has  the  potential  for  some  very  exciting 
opportunties  to  help  the  Army  improve  the  way  it  conducts  its 
maintenance  training  business.  The  potential  changes  would  not  be 
easy  to  implement,  but  they  could  be  brought  about  in  modest,  and 
measured  stages.  Working  with  the  test-bed  course  and  a  few 
carefully  selected  tasks,  it  should  be  possible  for  the  TTFA  to 
demonstrate  how  ideas  for  improving  skill  acquisition,  retention 
and  transfer  might  be  implemented. 

4.2  Testing  ideas  should  be  an  important  part  of  the  TTFA 
program.  While  the  recommendations  that  have  emerged  over  the 
past  two  years  are  based  on  scientific  evidence  and  have  worked 
elsewhere,  they  may  still  not  be  appropriate  to,  or  effective  for, 
the  Ordnance  School.  Those  ideas  which  prove  to  be  effective  in 
we  1 1- de s igned  evaluations  can  then  be  extended  to  other  parts  of 
the  63W10  course,  as  well  as  to  other  courses.  The  potential 
payoffs  for  even  modest  successes  are  large,  given  the  $22,000  per 
student  price  tag  for  the  63W10  Course  and  the  $15+  billion  spent 
yearly  by  the  Army  on  maintenance. 

4.3  Implementing  the  proven  ideas,  particularly  those  based  on 
the  functional  context  training  principles  of  PAM  600-11,  could 
involve  major  changes  in  course  design  and  management.  However, 
the  changes  could  be  phased  in  gradually.  For  example,  if  BKS  and 
task  training  were  to  be  integrated,  the  first  step  might  be  to 
reschedule  some  of  the  annexes  so  that  related  annexes  in  Phases  1 
and  2  are  close  to  each  other  instead  of  being  separated  by  5  or  6 
weeks . 

4.4  The  Skill  Retention  Model  should  be  used  by  TRADOC  to  develop 
guidelines  on  refresher  training  in  operational  units. 
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4.5  In  closing,  the  TT  F  A  -  A  b  e  r  d  e  e  n  has  been  a  succesful 
ground-breaking  effort.  The  three  partners,  USAOCS,  Training 
Development  and  Analysis  Division  (TDAD)  of  TRADOC,  Fort  Monroe, 
and  ARI,  working  together,  have  been  productive  and  have  made 
significant  contributions.  It  is  recommended  that  this  model  of 
R&D  coupled  to  implementation  be  continued. 
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